{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cifar10 Drift Detection\n",
    "\n",
    "In this example we will deploy an image classification model along with a drift detector trained on the same dataset. For in depth details on creating a drift detection model for your own dataset see the [alibi-detect project](https://github.com/SeldonIO/alibi-detect) and associated [documentation](https://docs.seldon.io/projects/alibi-detect/en/latest/). You can find details for this [CIFAR10 example in their documentation](https://docs.seldon.io/projects/alibi-detect/en/latest/examples/cd_ks_cifar10.html) as well.\n",
    "\n",
    "\n",
    "Prequisites:\n",
    "\n",
    " * Running cluster with \n",
    "    * [kfserving installed](https://github.com/kubeflow/kfserving/blob/master/README.md)\n",
    "    * [Knative eventing installed](https://knative.dev/docs/install/) >= 0.18\n",
    "    \n",
    " Tested on GKE and Kind with Knative Eventing 0.18 and Istio 1.7.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -r requirements_notebook.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace/cifar10 created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace cifar10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting broker.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile broker.yaml\n",
    "apiVersion: eventing.knative.dev/v1\n",
    "kind: broker\n",
    "metadata:\n",
    " name: default\n",
    " namespace: cifar10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "broker.eventing.knative.dev/default created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create -f broker.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting event-display.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile event-display.yaml\n",
    "apiVersion: apps/v1\n",
    "kind: Deployment\n",
    "metadata:\n",
    "  name: hello-display\n",
    "  namespace: cifar10\n",
    "spec:\n",
    "  replicas: 1\n",
    "  selector:\n",
    "    matchLabels: &labels\n",
    "      app: hello-display\n",
    "  template:\n",
    "    metadata:\n",
    "      labels: *labels\n",
    "    spec:\n",
    "      containers:\n",
    "        - name: event-display\n",
    "          image: gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display\n",
    "\n",
    "---\n",
    "\n",
    "kind: Service\n",
    "apiVersion: v1\n",
    "metadata:\n",
    "  name: hello-display\n",
    "  namespace: cifar10\n",
    "spec:\n",
    "  selector:\n",
    "    app: hello-display\n",
    "  ports:\n",
    "  - protocol: TCP\n",
    "    port: 80\n",
    "    targetPort: 8080"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment.apps/hello-display created\r\n",
      "service/hello-display created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f event-display.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the Kfserving image classification model for Cifar10. We add in a `logger` for requests - the default destination is the namespace Knative Broker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting cifar10.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile cifar10.yaml\n",
    "apiVersion: \"serving.kserve.io/v1beta1\"\n",
    "kind: \"InferenceService\"\n",
    "metadata:\n",
    "  name: \"tfserving-cifar10\"\n",
    "  namespace: cifar10\n",
    "spec:\n",
    "  predictor:\n",
    "    tensorflow:\n",
    "      storageUri: \"gs://seldon-models/tfserving/cifar10/resnet32\"\n",
    "    logger:\n",
    "      mode: all\n",
    "      url: http://broker-ingress.knative-eventing.svc.cluster.local/cifar10/default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inferenceservice.serving.kserve.io/tfserving-cifar10 created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f cifar10.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the pretrained Drift Detector. We forward replies to the message-dumper we started. Notice the `drift_batch_size`. The drift detector will wait until `drift_batch_size` number of requests are received before making a drift prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting cifar10cd.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile cifar10cd.yaml\n",
    "apiVersion: serving.knative.dev/v1\n",
    "kind: Service\n",
    "metadata:\n",
    "  name: drift-detector\n",
    "  namespace: cifar10\n",
    "spec:\n",
    "  template:\n",
    "    metadata:\n",
    "      annotations:\n",
    "        autoscaling.knative.dev/minScale: \"1\"\n",
    "    spec:\n",
    "      containers:\n",
    "      - image: seldonio/alibi-detect-server:0.0.2\n",
    "        imagePullPolicy: IfNotPresent\n",
    "        args:\n",
    "        - --model_name\n",
    "        - cifar10cd\n",
    "        - --http_port\n",
    "        - '8080'\n",
    "        - --protocol\n",
    "        - tensorflow.http\n",
    "        - --storage_uri\n",
    "        - gs://seldon-models/alibi-detect/cd/ks/cifar10\n",
    "        - --reply_url\n",
    "        - http://hello-display.cifar10\n",
    "        - --event_type\n",
    "        - org.kubeflow.serving.inference.outlier\n",
    "        - --event_source\n",
    "        - org.kubeflow.serving.cifar10cd\n",
    "        - DriftDetector\n",
    "        - --drift_batch_size\n",
    "        - '5000'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "service.serving.knative.dev/drift-detector configured\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f cifar10cd.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a Knative trigger to forward logging events to our Outlier Detector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting trigger.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile trigger.yaml\n",
    "apiVersion: eventing.knative.dev/v1\n",
    "kind: Trigger\n",
    "metadata:\n",
    "  name: drift-trigger\n",
    "  namespace: cifar10\n",
    "spec:\n",
    "  broker: default\n",
    "  filter:\n",
    "    attributes:\n",
    "      type: org.kubeflow.serving.inference.request\n",
    "  subscriber:\n",
    "    ref:\n",
    "      apiVersion: serving.knative.dev/v1\n",
    "      kind: Service\n",
    "      name: drift-detector\n",
    "      namespace: cifar10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trigger.eventing.knative.dev/drift-trigger created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f trigger.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the IP address of the Istio Ingress Gateway. This assumes you have installed istio with a LoadBalancer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLUSTER_IPS = !(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')\n",
    "CLUSTER_IP = CLUSTER_IPS[0]\n",
    "print(CLUSTER_IP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are using Kind or Minikube you will need to port-forward to the istio ingressgateway and uncomment the following cell.\n",
    "\n",
    "```\n",
    "INGRESS_GATEWAY_SERVICE=$(kubectl get svc --namespace istio-system --selector=\"app=istio-ingressgateway\" --output jsonpath='{.items[0].metadata.name}')\n",
    "kubectl port-forward --namespace istio-system svc/${INGRESS_GATEWAY_SERVICE} 8080:80\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CLUSTER_IP=\"localhost:8080\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tfserving-cifar10.cifar10.example.com\n"
     ]
    }
   ],
   "source": [
    "SERVICE_HOSTNAMES = !(kubectl get inferenceservice -n cifar10 tfserving-cifar10 -o jsonpath='{.status.url}' | cut -d \"/\" -f 3)\n",
    "SERVICE_HOSTNAME_CIFAR10 = SERVICE_HOSTNAMES[0]\n",
    "print(SERVICE_HOSTNAME_CIFAR10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "drift-detector.cifar10.example.com\n"
     ]
    }
   ],
   "source": [
    "SERVICE_HOSTNAMES = !(kubectl get ksvc -n cifar10 drift-detector -o jsonpath='{.status.url}' | cut -d \"/\" -f 3)\n",
    "SERVICE_HOSTNAME_VAEOD = SERVICE_HOSTNAMES[0]\n",
    "print(SERVICE_HOSTNAME_VAEOD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 32, 32, 3) (50000, 1) (10000, 32, 32, 3) (10000, 1)\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import requests\n",
    "import json\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "train, test = tf.keras.datasets.cifar10.load_data()\n",
    "X_train, y_train = train\n",
    "X_test, y_test = test\n",
    "\n",
    "X_train = X_train.astype(\"float32\") / 255\n",
    "X_test = X_test.astype(\"float32\") / 255\n",
    "print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)\n",
    "classes = (\n",
    "    \"plane\",\n",
    "    \"car\",\n",
    "    \"bird\",\n",
    "    \"cat\",\n",
    "    \"deer\",\n",
    "    \"dog\",\n",
    "    \"frog\",\n",
    "    \"horse\",\n",
    "    \"ship\",\n",
    "    \"truck\",\n",
    ")\n",
    "\n",
    "\n",
    "def show(X):\n",
    "    plt.imshow(X.reshape(32, 32, 3))\n",
    "    plt.axis(\"off\")\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "def predict(X):\n",
    "    formData = {\"instances\": X.tolist()}\n",
    "    headers = {}\n",
    "    headers[\"Host\"] = SERVICE_HOSTNAME_CIFAR10\n",
    "    res = requests.post(\n",
    "        \"http://\" + CLUSTER_IP + \"/v1/models/tfserving-cifar10:predict\",\n",
    "        json=formData,\n",
    "        headers=headers,\n",
    "    )\n",
    "    if res.status_code == 200:\n",
    "        j = res.json()\n",
    "        if len(j[\"predictions\"]) == 1:\n",
    "            return classes[np.array(j[\"predictions\"])[0].argmax()]\n",
    "    else:\n",
    "        print(\"Failed with \", res.status_code)\n",
    "        return []\n",
    "\n",
    "\n",
    "def drift(X):\n",
    "    formData = {\"instances\": X.tolist()}\n",
    "    headers = {\n",
    "        \"ce-namespace\": \"default\",\n",
    "        \"ce-modelid\": \"cifar10drift\",\n",
    "        \"ce-type\": \"io.seldon.serving.inference.request\",\n",
    "        \"ce-id\": \"1234\",\n",
    "        \"ce-source\": \"localhost\",\n",
    "        \"ce-specversion\": \"1.0\",\n",
    "    }\n",
    "    headers[\"Host\"] = SERVICE_HOSTNAME_VAEOD\n",
    "    res = requests.post(\"http://\" + CLUSTER_IP + \"/\", json=formData, headers=headers)\n",
    "    if res.status_code == 200:\n",
    "        od = res.json()\n",
    "        return od\n",
    "    else:\n",
    "        print(\"Failed with \", res.status_code)\n",
    "        return []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normal Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVlElEQVR4nO2d2Y9kZ1LFI/elKpfasrauraurymV3224vjTdspIExA4MGNIDghSd4QuKBf4c3xAM8gIxlazR48ODxYHsse9ruvauXqq7urn3Jyn279/IPxAnJPDAhdH6P39GXeTPznrxSxBcRsSiKhBDij/hv+gIIITo0JyFOoTkJcQrNSYhTaE5CnJK0xH94/2cwlPvkztdw3+HmbXU9CPDbTc4/A7X55XWojUzNQy2b099v4+ZncM+j+9eg1q83oJYwPltxpAS1ZDavrl95822458Iq/q46ZydQu3njKtTCsKeu9/oduOfWzetQq1WPoNbtdaHW7yXU9ZPjFtzTaOFrHAT4vSYmRqE2MjoMtSCq6+/Vh1uk08ZZkff+7acxbZ1PTkKcQnMS4hSakxCn0JyEOIXmJMQpNCchTjFTKbVTHJYfK+MwdDQxqa8ni3DP9Px5qAUhjlHHQxxiD1sDdb1zegz3RG0clp8dr0Btfu4C1OYuLEBtZvacul6p6N+hiEgqlYHaoKynZkRE5s5N4X0DPZXS6bThnuopTi0dHeF7J5nOQk1ieiplZAx/5uwQvsaz2inUMll8+4eRfu+IiKSS+rXUzqpwT6/73QtM+OQkxCk0JyFOoTkJcQrNSYhTaE5CnEJzEuIUM5UifZzC6HWx1mrpYfnF1Vm4p9Fs4vcyKiNGx42Kj5T+37Oysgr3vPHaK1CbndTTHiIipdIE1PrJAGr5rB6WTxqR99gAh/nbTZze6Bq/Zz6np2BGyjh9tHz+Wajdvn0XahLD19Ht6qmxUnEE7kml8Vud1fahFol+n4qIhCH+AU5P9Xu13cIVMP+bVl18chLiFJqTEKfQnIQ4heYkxCk0JyFOMaO1A+PQc2yAI5CZdE5dPzvCfWXGpnAkdP45fKi8MjcDtRQK4xnNXvoDHBm+s4sPzLceHuLXjOOo4N3r36rrr67jSOjbV16FmtXBv1Y7g9r2ox11PZ3Ch9TTaVzIMD6BI/Pbj+/h1wQ9lRptHM2v1fB9lUyp7XlERKRYxEUC7TYuqAhAsHwwCOGeTMYIKQP45CTEKTQnIU6hOQlxCs1JiFNoTkKcQnMS4hQzldJt4fD1cA6H2Iuj+iHwl154Ee6ZO78Ctbpx0Pvuw8dQq7X0cHijWoV7jqs4XbK7h/vRFI2D7xLHB6I/+Jd/VddTf47/N995/S2opVI4TTQ1hdNOEunpiOqpPnpAROTXV/HoiqTR52iogFMwg0BPBfUaVbgnYTxirJELQYBTXMcnOD0TFz0Fk0xiO5XLuEADvw8hxCU0JyFOoTkJcQrNSYhTaE5CnEJzEuIUM5WSyaSg1k8UoNbO6VOBN2u4yuWbX34JtZNj3Bfn6Q7uEZNK6BUJqTiuHuiCsQQiIp0O1qYn8Fd5sPcIakVQrVCv1uCejc1NfB3T41BLpfA1Ts/poxpmwLqIyPYeTmPdvY61yjROO21tgxRGH/9mYQ9rgdG/KZvG6Z5MEt/77Y7+msUiThElwQgHCz45CXEKzUmIU2hOQpxCcxLiFJqTEKfQnIQ4xUyl5PN4uvJBFVeK3H+sh9Fv3bwB98SNMH9gjH5o13HlTAKkTNpdnKao1rFWN0YdbD25DbWhHE47rS2v6YKR0vnvT/8LagtLS1BbXcNjKMbG9KoJa/pzqYjTA/EBbibW7OJnAhpp0K7i6pggwE3ZsjmcEmnU8GsWjcqZTFafvt3rWSNKcMMwBJ+chDiF5iTEKTQnIU6hOQlxCs1JiFNoTkKcYqZSyqO4wuH+4w2o7W7pVRP5FG50ddbEzbMatQOoxUJckVCt66mPahuH3pNGJc74JJ7ynCvgBk6ziy9AbQ6E5Te//RzuScRwmqUf4CqMwyPcvOzSpXV1/cLKebhnzqguGX7tMtSu3dmGWrejN47rpoyqFMFpjzDCKb+9PX0+jIhIOoPTRKURdB/gtF67jSuyEHxyEuIUmpMQp9CchDiF5iTEKTQnIU4xo7UPHuC+Pnce3Ifazu4DdT0wDqkXSkNQW1tZhNrF9YtQ2z3UI2SPDvF1TEzhw/4Ly/hQeWEMR3L3T/H7RUd6ZHv7EY5oHhojI4yB2PJ7q3pEVkSk2dC/qxAHfyXq4ajxzS9wtHll7UWoTc6W1fUvvvwF3LO3j4sV+n0cre208fWfGmMocsNldT2McES5aYw2QfDJSYhTaE5CnEJzEuIUmpMQp9CchDiF5iTEKWYq5YtffIQ3ToLeNyKyvH5JXc8ZbfPXn8WTrddWz0Et6OgHx0VEorieHmgKnlqcTOGJ3YlEGWr9AT4o3ayfQK3U00P9aMKziMj2AS4SyA4/xe9VHIHa+eVFdT0y/r/bVdwX586vvoFa1Mb3wcV3f19dv/Q8PoDf/gqnUh7c34JaPq+PDRERKZXHoCai55dqNfy7dLvsIUTI/xtoTkKcQnMS4hSakxCn0JyEOIXmJMQpZirl4DFOOVx+4Q+hlsnovWVGcdZDpmdwH5gToxX/4/s4TdEL9fRGPIZLLRJJY0pyhHsgycAaJ4H7x0SB/n7DJdy/6biBKxziaVzdE0Y4PSMCNPx1yHAW/2aLM3NQyybwdcRF7/t06SKuCCqXy1B7v/0fUNvbxamP2coM1IKY3oPKmhxeq+F0D4JPTkKcQnMS4hSakxCn0JyEOIXmJMQpNCchTrEnWw+PQi1lROWrVX18Qma0DPe0Bjhm38HTEyQ3gqdGZ8IYeEGcSomMb6TTx5UF2RzeGDfGJ4Rxfd/wGA7lpyOcPkrkcOVJlMa5rDCmf7ZYgFMz8QT+zKmhNNRyw1gbdPW02fHTfbhnbAiPhfjRH7wLta++3YJaw2j+1ekequtdY+RCuVCGGoJPTkKcQnMS4hSakxCn0JyEOIXmJMQpNCchTjFTKdPzuBIgFse+7nT0E/j7Nfx26TKuwugPcOg9lsKTqNsNvcKhH+FrTyZxo65BAmv5Iq7QqIxVoRad6OH3njHjIxbi68/lclCLG1VBaAJ0YEzKjqeM5moJfI2NJq4yQpPKM8b9VjvEaZZcHqcD3379eajdffAIajdu7anrjRquFkobjeMQfHIS4hSakxCn0JyEOIXmJMQpNCchTqE5CXGKmUqJYjhUbo3zbtX1UHnGCPPXa0ajrg5urNWq4bB8ChSlFIZwSmRiBIfei6O4QmOijD9bkCxBrZ3Rv8eTBVyV0g12oSZG5UwwMKpjQAVPEMfVQjEjlVIexdUxYWBcI7ivSiX8/aZjuESqWq9CLerrqTYRkRfXp6BWLuj3zwcf4GZih/u4WR6CT05CnEJzEuIUmpMQp9CchDiF5iTEKWa0VozoXjLEWgmc8Z0rgfCpiDxzvgy14SyO1CVi+P+lWauq653WGdyTG+pDbW0FR3LnFvD07XhqAWqNalV/velpfB2beo8mEZHiKD5gPTqCD+cnk3pxQWj0ioqMg/TZoTzUBh0c6Y+D90tZhRaCo/lj43h6daOFo8bNqn64XURkdkLvWfTHf/R9uOe9D38GNQSfnIQ4heYkxCk0JyFOoTkJcQrNSYhTaE5CnGKmUt55/WWonX/2BajtPH2qrs/O4FTE6soy1KYmKlBLRDg9UweHnrvG4fBYHL/e8BA++D48jFMYiTROBaVASqrd1Fv+i4i8dBGnZhZXF6HWD3GaKAL/04MQpz2iBP6uEsaU534H52dCcPA9nsTPkVgWX4cY+7p9/H0kE7g3VdCrqusTRtrmrd9+FWoIPjkJcQrNSYhTaE5CnEJzEuIUmpMQp9CchDjFTKW8/PwzUHvuMk6ltC/qaZGhEq6KwJ1qRKIYDpXHjZD36JDeB8aYxmD+W4VgVICIyMDoqSRGyL7b1ccxLF+Yh3tyaZzSaTdxxU0EpmiLiEhM1yKjP08YYS0wfrPQKHXpgenQQWhM2E4a94fxi9aPcUrt0eZjqL351mV1vdXH/azyVroHwCcnIU6hOQlxCs1JiFNoTkKcQnMS4hSakxCnmKmUnFWFkcUjDYby4GWTuCOU1UgqZqVSrJB9pKc+wj5OiVjpAWua98BIBhmFLhKBBmXDZVzBMwjwewWhNb4aX0gk+gTruHXxAdaCJE5xRWL82KCpXCzEE7YzxmdOBfg3G+oYk7n39ZSOiMjhQ32S9rk13OTtKI5HPyD45CTEKTQnIU6hOQlxCs1JiFNoTkKcQnMS4hQzlVIo4XB+ZFSDtLp6ODzq4pkWXbBHRKTZaEKt18f7ul29GmQwwKmIvlFB0jfeq2XM3Wg1cbXCAFS6FEbxNOxCqQy1cmEcatm0Pg9FRCRAs29ixlwTwVqhgBueHR/g77HT1lMOYYgnZccEf64wwPdcEUyoFhFZmJ+EWrul34+R0QytVMBpSQSfnIQ4heYkxCk0JyFOoTkJcQrNSYhTzGjte+//BGpB6lOonZ7qB4MbZ0dwD5poLGJHcvf39fcSEQnAafpRY7zDyPgY1DIJ/HU1T6pQ27h3G2q1hh6dnFvCIxcSKRwpLxbw9S8t4b5E5+b0fktL52fhntEMPvheyOJrDI1eUpLQD6P3AxwJTRgjFxLGNU4uGpHtIo7k9iP9EH4CB41ldNT4zAA+OQlxCs1JiFNoTkKcQnMS4hSakxCn0JyEOMVMpXz088+gVj63BrUo0NMDVz/7OdyzcA73Xxkfw+mBp0/2oDYAfWfyo2W4pxfHh+L3n+AW/d+78jrUXnz+Oai1uh11PW5Mht7cfgS1jXsPoHb9xlWolUv6VOYf/+mfwD1vPrcKtbQx8+Lc9BzUeiCVYk0ct/o+9UFvJBGReNLoS1TGB/dzoJdUmMApP5xYwvDJSYhTaE5CnEJzEuIUmpMQp9CchDiF5iTEKWYq5c/+8q+glqmsQK1V19Mb965/C/dMT+HwetwYg5DL4tP+vVBvqb96EV/7yDSuWGmN4z42P/zB70ItX8hBrQlSKcbkBBmAMRMiIp2B/noiIgcHJ1B7tLmjrufz+Pvde3IMta2b96AW7+BrfLh3oK5f+f4rcM/C4gzUrGqWeNYoI0nhNEsM9QqK4T3pmDW7XYdPTkKcQnMS4hSakxCn0JyEOIXmJMQpNCchTjFTKZk09u7GnRtQq53pqZTIqh7o4RP9DWMcgzX1OpvRawH6LTwe4ewQX+P+Nq5K+clPcTO007rxfo0zdb1QxCmM0ggekzFkNKZ68kRPl4iIVMb1Rl7ZIk4tffoh/swn965BLejhkRf39/SGbU+MkRYr6zg1VirmsTaCR17k8rgqpTSk31epLJ6Unc/j3wXBJychTqE5CXEKzUmIU2hOQpxCcxLiFJqTEKeYqZT6MW6e9fG/fwi1x3tP1PV4X68SERG5dq2GL8RIlwwGuOpAQCXARx98DLekUzjk/eLll6DWSxegVuviqdcPt/UqjONjPF+l18EVDjt7W1Db3MKv+crll9X1v/vbv4d7vvzic6gNznDFSs2YcN4WPZX18Cucxvr0612oDSVx2iaVxqmPRAbfBwWQSjm3sAj3/OjHfwE1/Zvnk5MQt9CchDiF5iTEKTQnIU6hOQlxihmtnZ6chtrK4hLUItGjiUlj1EHCiMjGE/g/JALTq0VE0tkhXUjhQ80zM3iS8++8+y7UCnnjgHUW9x66dUPvq7RxH49VmJpdhFrHGIOQyOFrvLFxR12/tbEB9+QX16G2s4M/80gZa5W03tcnP4z7MJ3s4fEUx0/vQ+3wCE9F7wRGkQZo8LRbxXZ643tGUygAn5yEOIXmJMQpNCchTqE5CXEKzUmIU2hOQpxiplJODnH7/td+6w2ovfHOO+p6JoMPGieNdIk1jiE0RhMkRH+/fg+3zW/38CH14yebUDvp4APWJ0f4e3wIUiY7B7joYLiCxw9IBqeJYmmcSukN9MPoH33yS7hnYfkS1OZGcUoqG8e3XR4UHnQ7uIfQw9pNqA0XcC+mIMJFE3un+nR2EZHx8UV1vdXH9+LHn3wJtb/+G33sCZ+chDiF5iTEKTQnIU6hOQlxCs1JiFNoTkKcYqZShowW8sc1PJ346rWv1fVKBVcjTFbGodbv4zTF6WkVagImKCdD/HqzSzhNMTeC+wQ93cB9bJoN3DOnMjmlrufHynBPwpjm3Wrj32V6eh5qezt636ejY31chIjI9IwxJsMYvdHo4u9fkvo91w9x+iuTA9VHIpIxqp16x4f4OuJ6nyARkUlQFdTr4pEixteBL+G7byGE/F9AcxLiFJqTEKfQnIQ4heYkxCk0JyFOsSdbp/Ap+26nCrXPPvtPdT3q4zB/MY8bOPX7uHqg08YjHpLgv2dhcQ7uufjas1BbnsdplupjPRUhIrJ3egS1dE5PHSyP6SkWEZHDQ1wxcWntItSeu7QGtX/+p39U15OiN9wSEek38e/Z62EtGuC0iGT139oaj7C4dB5qB4/v4veK4yqp3BB+v/X1VXW908K/y9w0nhCO4JOTEKfQnIQ4heYkxCk0JyFOoTkJcQrNSYhTzFRKq42bXYnRdOvdH/xQXQ97uIohYaRLwgCndKKEMZ04qacBskO40dVeFadm6lU8N+Skja8/lsVNt+5+81BdP/4cV0ycX8IpkVcvrECtZ1Ss5NJ66iAyKoKsCph4At9aYNSIiIi0QzBnJ8Df78I5nErpNPCE7WeLuJrly6+vQm3nkZ6eaTfx/R21TqGG4JOTEKfQnIQ4heYkxCk0JyFOoTkJcQrNSYhT7AZfw7gioWQ0LCpM6Kf2u13c6Cpr/E+kY/g6ohyuZsnk9X1hB1cP1Os1qCXyuLFWZbkMteU8rkq5twnGy8dwiihlNF57ursNtbFx3GANab02Tg90u7j5V9OoWOka1Rv9rp6+S2Zx+mtyZgJqj3bxaPn9bfDdi0ingT/bg5vfqOtjY/g6opFRqCH45CTEKTQnIU6hOQlxCs1JiFNoTkKcYh98r+OD3hJiX6diw+r6/j6OgN27tQW1bBJHZNOlMtTGwfiHmfES3JM0DvSPlcagZpzNl04bH3quVPQI8OwMju7t7uGp1xsbt6G22FuCGoqk1+v4N2u1cCS0doaj3la0NujphQeJDD6kfvMGHuVhjUioVCahNvs87sVUmdD3jU/gvk9Z4/oRfHIS4hSakxCn0JyEOIXmJMQpNCchTqE5CXGKmUoJjZb6ccPXyb5+aLtojHf4+otPoLa3jw+Ox1L4EPiVKy+r62+9/grcc3aGUwfXfv0rqDXBFG0RkY3tx1B7uLWlrrdbuH9TFOEmPNkiPnxdq9WhVgcjI5o1nAYyWgFJMoHVUgEfYp9Z0tM9I2PTcE9lBqcwZi5fgtqo0UMobfWmQppRrCDRd38O8slJiFNoTkKcQnMS4hSakxCn0JyEOIXmJMQpsSgymgERQn5j8MlJiFNoTkKcQnMS4hSakxCn0JyEOIXmJMQp/wNMLfgMByU9KwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'truck'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx = 1\n",
    "X = X_train[idx : idx + 1]\n",
    "show(X)\n",
    "predict(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Drift"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to accumulate a large enough batch size so no drift will be tested as yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl logs -n cifar10 $(kubectl get pod -n cifar10 -l app=hello-display -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now send 5000 requests to the model in batches. The drift detector will run at the end of this as we set the `drift_batch_size` to 5000 in our yaml above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bfd434d2daab4f7989e3794e1b181b22",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=50.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from tqdm.notebook import tqdm\n",
    "\n",
    "for i in tqdm(range(0, 5000, 100)):\n",
    "    X = X_train[i : i + 100]\n",
    "    predict(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the message dumper and extract the first drift result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Drift False\n"
     ]
    }
   ],
   "source": [
    "res = !kubectl logs -n cifar10 $(kubectl get pod -n cifar10 -l app=hello-display -o jsonpath='{.items[0].metadata.name}')\n",
    "data = []\n",
    "for i in range(0, len(res)):\n",
    "    if res[i] == \"Data,\":\n",
    "        data.append(res[i + 1])\n",
    "j = json.loads(json.loads(data[0]))\n",
    "print(\"Drift\", j[\"data\"][\"is_drift\"] == 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's create some CIFAR10 examples with motion blur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from alibi_detect.datasets import fetch_cifar10c, corruption_types_cifar10c\n",
    "\n",
    "corruption = [\"motion_blur\"]\n",
    "X_corr, y_corr = fetch_cifar10c(corruption=corruption, severity=5, return_X_y=True)\n",
    "X_corr = X_corr.astype(\"float32\") / 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAS50lEQVR4nO2dSY9k13GF474p58yaeiQI2YZkwIZX+mv6Gf4XXupPCF4aIAhvKBgy3KTIZrO6uyqrMiuHN18tqOU9hyAXZqBxvuULvDHfyQfEuRERYowmhPBH9mtfgBAijcQphFMkTiGcInEK4RSJUwinFCz4H3/4PUzlZiGQPdO7DeOA96BZYxwL5Dqijel9yJnY/1VGYiHga8xJLMvS1xjJPY8jjg0kNo74zoeYjg0R3/OQvvS/74djjBEclJ0L7WNmxl6rccT7tSRW9+mDdr/wd/n3P75JPnx9OYVwisQphFMkTiGcInEK4RSJUwinSJxCOIVaKZGYDmMkuW1kA4B0vdlP2BssSGLIwQgZsV+w2wOMmR/JmbVErBS4G/MAyEUGYsFE8l8cwaswBGKlkHsmToRF8u5E9I6QxxHYNyb8gvf0J2PI/iKXwf27JPpyCuEUiVMIp0icQjhF4hTCKRKnEE6ROIVwCrVS+rGHMVppAfwN9k+AKkjMuPXBQFfIqhECeSQh5GQ/YpcY82fAM4742bOc/Rjx9fdZBWPtWCa3Hxt8rr4nVS49vmdW3VMg64bYLzlzUsh7FYnNwt9HcB3E12NVKQh9OYVwisQphFMkTiGcInEK4RSJUwin0GytjS0Msd49ASxezjOy8Jpl3Eiylqyltwh75rD/JPxIsgxna2fTdLbTzGwAPWfMzLo2nQ4NAV/HGPC5mjiBsUOHj/lh3yW3twM+Vz/gh58HfB3ZmD6XmVkGstTTEj97G88wNClJlpekebOc3BtIl9N8rBa+C/HpIHEK4RSJUwinSJxCOEXiFMIpEqcQTqFWSkZ61eQknZ8Dy4G0ozEjmXLam4X1OUKnIpZIOZnB2KTCsdXFGsa2d+9hrAeuwhCJXTJOYez2Ea9U/+GhhrGxTF9/MdvAfQZiRfQ9seF6fI25pR9IaI9wn/UM/y7DgO+5IgvfC/KuZnn6jcwNFytkzPND+/zsPYQQ/y9InEI4ReIUwikSpxBOkTiFcIrEKYRT+DiGfA5jLelVE8AE6xxMcTYzK0iPGDpDm+4HqmNInvz62Q2MFZMFjN3fPcDY4xnfwbFbJrdvdzgt//bDFsaeWnxv1eIaxpbLF8ntxRxbRHTmQoetlK7Gtkh93ie3zybYWjqQqpTQ4gqYOXn7l7jdksECGdJXi00qR+jLKYRTJE4hnCJxCuEUiVMIp0icQjhF4hTCKdRK2TZ4tX8kZSSnw11y+2aB08k3JHddsJnSZAJ0yNPXWFb4tpsGp+W3uxOMvScVH3d7bItsD+nrf7/FlRtPLbYVZutLGCs32CbKZ6vk9vkKWylsSsZ+9whjXYef1WyVroLpGmy/7J6eYGyS42cFp2ibWRlwRVYFqnGKDL+LbAQIQl9OIZwicQrhFIlTCKdInEI4ReIUwikSpxBOoVbK90943sXTcQdjaLX/bIbT2iMZlhJIIybWNAzNu+hJxcTdPb6v+xM+2QOJHXpc3XMENxAX6WoVM7PpkjQom+JzVXMcm83Sv/W0IJUWA7aIKmJFRGJltV26iuRY43Ode/xenUkFTLXAv1lLKqg6MPtmUmBr5pcMZ9eXUwinSJxCOEXiFMIpEqcQTpE4hXAKzdaec9yK/zDgjGdVpjW/XOPjZRlevDyySchgcbuZ2WDpLN6JLBw/DHix/8cjzkCWpD/PqiIjDY7pLOR4xs/33ODnEcj/LUsYVuAxdif8u7Cig4JMhm7IIvDjMZ1dPTc4W9t25Doifj/OLVncTootSjCOYYIeopll6iEkxKeDxCmEUyROIZwicQrhFIlTCKdInEI4hVopxewCxq4rPJrg5Sqdhp4vsT2QoxHPZja2eL/B8OL8wdL9b7J1evSAmdnE8H3943Mcq3v8P9f02FbYn9KjFcKIrYNAet8E4pecSK8da9L9kdZLfM/TKZ6wfa5xD6S2w/eG+vqwaeRFSQoqiO107rCVMiW2yAkUTlQdfgfmUzLfAaAvpxBOkTiFcIrEKYRTJE4hnCJxCuEUiVMIp1Ar5fnr38BYe8a9WV4/T6eNq/4W7hOPOM3f1Lh9/2LzCsbmq39Ibr/+7F/wuchk6OMTvue7B9x7qOuwrVCGdMo+IxUfkwpbBw8PjzCWk8nL2SJdjXNzfQX3WZEqo37E5+qJFYQqbsqKVIlMsE1xjqT3ELFZ8hM+Xw6aZBUttl/QaBCGvpxCOEXiFMIpEqcQTpE4hXCKxCmEUyROIZxCrZTlBjetYuMCikl61X6IuIIkZiyGKyOqBa4wuXn1u+T2c01uO5LYiKdel2SCcp7hqprNJj1R+tjic5322HZiVSksOAO/5/MXL+E+6zWeel232KbYH/G9Gag+qSr8fvSkoqmc4IZtLatY6UkztyZtswQw8drM7NxgOw2hL6cQTpE4hXCKxCmEUyROIZwicQrhFIlTCKdQK6WscBq6JtOhQ0gftizx8Xow18TMrMxxGv30lG5MZWZ2+y5dBTOGtH1hZjYhk6G7HlsA0fDzWK1xIyyr09UbswOZ1tzgKgzWPOvy4gLGXr/+LLn95hm2quoaP4+2J9UgNX5WaKILm4nD5pBkA353CtKkzkg1SxvT970/4uqp2OGKJoS+nEI4ReIUwikSpxBOkTiFcIrEKYRTJE4hnEKtlJ5UFrB5HdNJ2o7IRmwBnGqcumbzLrrzPYyVQ/o6YoFT3tkJWxhkXIfdvMLVGzWpcBgP6f/H9RFXfIzGGknhn/Ti8hLGNhfpWE9GxJ/IPBRml7DmXwNobBYH0jCsx9doAX9/Fiv8jMOA7y2CCpOB2EeRPEeEvpxCOEXiFMIpEqcQTpE4hXCKxCmEU2i2tipwuCjwAvEWLIrf7/BC6WNDpjX3OFM3DjjzOjT75PbTCe8zWeIRAy9IP53LZ89g7OmEM3/3+/QzCQFntlcL/J+Kig7MzMoKHxNlUFsy/bkmGdmmxZnLnLxXAWS22wb/ZtUEL26vSnzPmZEMKr5tG0AvprHB72mlydZCfDpInEI4ReIUwikSpxBOkTiFcIrEKYRT+DiGGel9M+JU+fGY7usDXIMfDxdwf6E44j5BbPrA4QxsihIvHH92eQFjy4sbGMtKPJ5iJIuvz6d0zn5CbI+uw4UANWn7PyUjNNDU6/qERz90PbZSlit8ruMZ2yL9kH4eeYELEqZT/J7Op2TMx0B6QpFF/WORjs1n+Dcrfv5ga305hfCKxCmEUyROIZwicQrhFIlTCKdInEI4hVopM5KibklFwhjS6euhwOl1m+DUdd2Q0Q9kmnAGxj+sN7jypCAVE1fP0iMLzMwedgcYe/PmGxjrurStsNtv4T50nAGwRMzMsogtjBH0Ypov8ciCA6kIas5k/EDEJR9VmbZMApnKXWTkHSBjFcZmB2N5i3tTbUBB1oJUnozknhH6cgrhFIlTCKdInEI4ReIUwikSpxBOkTiFcMpPTLbGTbziiJfZz5fpdP6xxtUlYMCzmZn1E9LQasDHXKMqgUBKBHJcHfP1m69h7Pb2BxjbPT3C2PmctmBqMoIiK3GlhZFKi2rEdsQK2ADb7Qe4zw8fccVKQd6dkpRo7HbpY7LJ1rHDNtwQsaWzznHscobfuc0cvSPYtkFjJhj6cgrhFIlTCKdInEI4ReIUwikSpxBOkTiFcAq1UnIyZ6IgdkRZAM1XuMKhI/NLiimuIskGfB0DcA4Gw9U2t7cfYexx+whjTwdc4TAljZ/6IT2vI6+wXZITK6XMcco+Rjwb5K/ffZ/cfsaOji1XeFL29iE9p8bM7Pvv3sFYC6ZUMyfiYomf73qKrbZnK/xtulyS5nYxbd2g39LMLMSf/x3Ul1MIp0icQjhF4hTCKRKnEE6ROIVwCs3WsvXhwUgQLIqvyIiBgmSGo5FUHekfE8Elbnd4wfNhj7OuZIqDRZJOLAp8b3mV/gkC6H9kxp/V2OKfdPeE++KgxeiTGb6OPemb9PZ7XAjw8IgXzKOXbjnHv/P1Ao9q+HyDn8fzNS4EyA33aWrBOIyMzAYZ2BRtgL6cQjhF4hTCKRKnEE6ROIVwisQphFMkTiGcQq2UpsaWQzHBC4OLkG49PwO2gZnZkfSIqYk90IOF0mZmEaTl1xe4v82WLG4/g0nZZmaTCV6M3g34P/Dli9fp481XcJ/7+zsYGyLO50/naxjr2vRi7vt7bC198/Y9jD3u8YLzgfQyms3Stsg1moFgZhuyRn29IDZWhnv+9D2+xgb8nmyC+TjKShHik0HiFMIpEqcQTpE4hXCKxCmEUyROIZxCrZS6xtbBxRRbBz3oVVORKdR5jlPeQySjCQp8Hct12o4oAk6h37x8BWPbhwcYm85xf6TPP/8NjL0C5zucsBXBJlsPPb633SM+5iOo1Pn67S3cZ/uExyD0Eb9asxl+VovLq+T2DDspNpLYltgbH2py/SOeRN326Qok1kMoGilZAejLKYRTJE4hnCJxCuEUiVMIp0icQjhF4hTCKdRKOY24cdJiJNOmQ/qwI2lJ33bYAihLfB2fvX4JY7/9p7SFwUYnLDYXMPYq4OvoOpxGX67wOIkaPJNTh+2S84Dto4caV/e8vcPWwe1dulnXYcC2h62vYWgxxf4Gs1IimLB9mOBX9Vs0d8PMxiObKI1jw4hjdZe2Wdj7zc6F0JdTCKdInEI4ReIUwikSpxBOkTiFcIrEKYRTqJWy63HKftzhtPwcHLUl9stkimdyvHiGJyj/27/+M97vxfPk9u0ez/iw2QUMffnV/8HY6YSfR7zDU7t70FjrfMbHu3vE1TFsDsnhjK2gZpq2pAJpkFVUuLNWSea5WIGv4wyash3B/B0zs9DjWJ5jmyWSyhPWOI41UUOowZcQnxASpxBOkTiFcIrEKYRTJE4hnEKztd8+4CzjpsQLeZdFOraZ4+zvK7KA/fnVEsdu8KLy2SydTdz+gLOd//nF/8DY/36DpzV3LWvtjzN145DOGDZgcbWZWd3j2EB69wwlflY9GpWRkSxpjs/VZDhWs6IJcEw2Sb0EhRZmZpFkSUfS84eNVujBFPORZH8DeY4IfTmFcIrEKYRTJE4hnCJxCuEUiVMIp0icQjiFWilv3u9h7HqOdf3ZRdrCCIatmSVZ+L5ZYytlMsOxb9+newX96b++gvt8+ee/wljdYfuIrWuOIPVuhnvVDKTlzED+U4dA+unQY6ZT/QWxKXr2+hBLJyMWTAkWxU8neLF8RvrzkEdvkTwr4rIYWvc+nZMp2gWVWhJ9OYVwisQphFMkTiGcInEK4RSJUwinSJxCOIXmd7dHPBJgBNN9zczyMT2JesBuiV2tcD+aan4BY3d7fI1ffPV1cvt///kN3OdE7muIuLIgI1O7B5KX72L6+lnzftaNhk1QjqTSIkf2RoYtDBbLyAiN6QT/1pMK7AempZuZDaCyx8ysJ9+fjnhLFbH2rjbpiekVuedxwFVLCH05hXCKxCmEUyROIZwicQrhFIlTCKdInEI4hVopDWlM9dCQ8QNtOp1fXuGJxkOxhrHbR2yXNHfvYewv371Lbt83aavHzMxyYg9kJC1PJnOzqhQDlRHjyFr+41hGGknlpOmWoVhOxjFMcMM2VnnCmoa14J1jdklLrCpW3bMiU8yvr/AIkByYWfURj/kImmwtxKeDxCmEUyROIZwicQrhFIlTCKdInEI4hVopxB2wjlRvFKD64bHG/wUfD2TOxB1uNPbu/S2MbY/phmItqUYYWVUHsURYVcpI9gtgknNBbI+RdeoilSeRzBvJgGUSyBRq2tQssOoeNlE6/dK15GUMxP5aX2JLhMUOB2yLnPbb5PbhfIT7zGakJAugL6cQTpE4hXCKxCmEUyROIZwicQrhFIlTCKdQKyUvcEXCQPLoNbBZHk44hf6X7z7A2P0Bp6E/fMRWyruP6ZR3Q8a2t2QeCrI9zMxyNp49w/ZMDipd2Ej0wKwZUg1SlPj3jMC6YW2pIqm0GHtc+dOTe+tB9UlR4QqYi6trGJsvNzD2tMcW3f4ev4+xOSe3Z6T1GpvngvcRQrhE4hTCKRKnEE6ROIVwisQphFNotjaSjGxA433NbADZxP0JT7ZuuxOM3d7j7Orh+IRjp/QxO3w4I61q4NRlM7OqIj1/SKIOZWXJunErSdv/osIZWda7B41qKHK8T0ey3uea9Jgi79UcjEG4vL6C+yxWOCPLRlecRpyLHki2uWvT9zYtsZwOB7woHqEvpxBOkTiFcIrEKYRTJE4hnCJxCuEUiVMIp1ArhfkKbJFvBD1iGtIHhqXes8DS2syeSV9jx7wUZhGRBdtth68xkL5EJbAqMrJYPrBF8eTWqnIOYwWYKD2QfkVdi++5MHwh1QwvYl8t0yM7rtZ4lEc1xa/xidh3I3l3+haPABmAhXQEFouZWU/eD4S+nEI4ReIUwikSpxBOkTiFcIrEKYRTJE4hnBLo1GUhxK+GvpxCOEXiFMIpEqcQTpE4hXCKxCmEUyROIZzyNwUpyHZhrUkTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQ00lEQVR4nO2dS28k2XGFIzMrs54ki2Sz2TPjeUljCYIWkgD9ZwMGDHvhhQ0YNuCFf4C9MDyQMAt5hFG3+sEmi6xHVqUX9vKeM+4CpxVuf98yAzcrKytPJRDnRkQ1DEMAQD7qP/UFAEAZxAmQFMQJkBTECZAUxAmQlJELPn/+XKZyP9Qsb1VV7/cDj7iNx975Y9YdfzfMp1WPfCXv+VFUz8ixmri+vi6ekDcnQFIQJ0BSECdAUhAnQFIQJ0BSECdAUqyV0jSNjGGlPBbiPtrbe9y9H5wdIULDoNdU1mjR11gdZaWYT/o/bqUoeHMCJAVxAiQFcQIkBXECJAVxAiQFcQIkxVopLjX8oVopP8T3Os6cefzrcGc8CMvE3w9nl+hVlTnn+zayHhOsFID/JyBOgKQgToCkIE6ApCBOgKTYbO373wT+uGTJKPt8Zznqrv2wP8jYzsX6vYxtd+VY3/f6Og76fPPZVMa6kS6omIy74vHaPIs/xK98zLP/2HrhzQmQFMQJkBTECZAUxAmQFMQJkBTECZCUoze+f6gcmw73LX9cVGw4Nyv2Znv47qBX3m+1LfLwsCke73c7uaZp9HXMptpKaRr92FXifeF/lyOfU3PODCYib06ApCBOgKQgToCkIE6ApCBOgKQgToCkWCvFVgI8ss2SpbLAYXvwGAujP7x7pch6qy2Mu4e1jt2XLZGIiM3OVZiUr39UG7tkUq4giYjYmuqYsbmRcvrDu0+S+F9F5aoEFVm8OQGSgjgBkoI4AZKCOAGSgjgBkoI4AZJirZTNZitjzmapRPrdr9H/E8dWJLipzIqDa6xl7JKtaXa1MdUga1H1sRJVIhERL9/eydjrtyv9WcaeCfHd3L/3pNXRtyczGXt6cSZjVxfL4vHZdCzX+EHZ714R9H0cU8tyzCfx5gRICuIESAriBEgK4gRICuIESAriBEiKt1JMArg1TZoaYX0MJudtih/C9JGKyiWpxccdBl0xcdibeSJmbsjWNMLamFhdl+9j1+p5IovJ5J3PFxGxNbNS9iK2N/djZP7am1ZXrOwrvVCZd2Njtdk3jHFSHrvwxFk6x1Rd8eYESAriBEgK4gRICuIESAriBEiKH8cwmcvY3ma61CZqnSWtTMxtRq9NTEVc/yPfG0nHKhMbmTTeuCv/BNNOZztnY52tFQOqIyKicmMQRJbX1Q7UZkL1yKRyR42OqZDOk0fU7nepzbPjzumyvDLwuOla3pwASUGcAElBnABJQZwASUGcAElBnABJsVbKX/zlX8nYyVz3iLkUPWI+urqUaz66upCxi5OFjFlE+tpmtV2av3JWhPYcRm0rY63YIF7V2qaYGH9jb2IH81/ci5uyMZv9H9Z6LMTq/l7G7u91D6TtpnzOyhQrONvp9ETbgefLU73OPN+qF9bgeggdscmeNydAUhAnQFIQJ0BSECdAUhAnQFIQJ0BSrJXy5vVbGdut9biAWkxynjbaHjib6kqLE1OFYfsLib4zzqYI09+mMtffhE7nN3YiQPkLmMkPtkpnZ8ZCPGweZOztqmx9vHpzI9e8ePlKn+/2VsY2ZjL3IJ6dzlTAOFvv6sm5jLWmcmZunsfGPAePCW9OgKQgToCkIE6ApCBOgKQgToCkIE6ApFSuodW//Ns3MtiYZkadSFG7Rlet6ahUDboy4nDQMWWZuEZXg0mTu6qDg6kGqY11o9LyLl3vJn3vze/pxkncC2vs7l7bLysTG/a6iiT2psFaX17nijpMIVEsFtoSub7WVVKnC23PqAZl7hpd7PJsWQzz5gRICuIESAriBEgK4gRICuIESAriBEiKrUppBq3d1qTzG2UriIqDiIjV2qXsdXXMaqWbRd2JJlMbM1Dk5k5fx7d/eC5jr290FcbYVNUsREXF2aluanZxvpSxqyfaHnjyxDRROy9Xb1yc6SZYgy2dMSFjs6jJ4u6zBtP8qzE+y2Q8lrHeDgMqx9wMGOulCHhzAiQFcQIkBXECJAVxAiQFcQIkBXECJMVaKX/zt38nY+NWLz2Zl62DpUnLLy907ORUz7to5+W5LBER47p8HXdvtDXz3Uttl/zHb76Vsd9/p9eNx3pWyrmYA/PMzJUZPtPWweVC38d5o5uQLUbi93TNxEyVy3a71et2O71OxAYz4WYwFUG9sW32r5wFo6uCZvNp8fjUNgXT78EnpyfF47w5AZKCOAGSgjgBkoI4AZKCOAGSYrO1//rvX8vYYqYzU2dn5c3cz66fyDX7Wm9G7w86u7eY6UzuZFrOqp2a7N5Hz/Rn1ZXO4H31xRcydmYmc19fle/JJ8+eyjWXZiJzZ+ZT7O71GISv//CiePz1zRu55vWNznrfmoKEjZheHRHRiwywGq0R8T2Twzu9uX08MQUJ5jc7vywXCSyX2jkYqWx4RPzo2VXxOG9OgKQgToCkIE6ApCBOgKQgToCkIE6ApPgeQge9eXk21u3qT2bl09YH3Z/n+X9+I2M3btPwpbZnzoVNMTX2y1efP5Oxn3z5iYw1ZsSD2/RciY3lDw/6Xv32m9/I2OtXr2Xs7a22N27vVsXjK2O/rLd6uvne9ItqzJTqyaRsfUyFLRYRMRtpu2Q6MetOyhvOIyKmC22lVKNyAcGm13bgeqeLBBS8OQGSgjgBkoI4AZKCOAGSgjgBkoI4AZJirZTuoFPl10ttpfz4R58Wj7c6gx63b7QFMOt075vLE30dFydly2RhxhnMTXq9MdUPezMSQE2Njoh4eyssjK22Um5MVcfGWDqTC207za8+Lh63FpGZ2G2mdURjKjS6rhxrzb3vOlOVYtaF6Uu03enf7Fb8Zi9f6mf4cNA2S/z6F8XDvDkBkoI4AZKCOAGSgjgBkoI4AZKCOAGSYq2Uzz/7TMZ++YtfydjPfv7T4vGm1vn1g2ntPzap8olp4NQqC8a02j8YC6Af9DXuxUTmiIh2YsYxTMtNoWZL3eDr6Z+ZMQg7MzXaNDarhWXiJkof9jq2t6MatE3x8FCeRr5el49HRNzcvjHnK9seERFrU/njnscQtlllnm9XmaTgzQmQFMQJkBTECZAUxAmQFMQJkBTECZAUa6U8/fQnMjY9L1cxRES8vBUpdmNT1Gbqct2bcpY7nfIexIyVg6kg6Y0lstvrhmcbaw/o2GZbvsZ701irN82ietNkyk2ivl+XP29tJlRXpvTkYBp8KbskImItYv1OX8fIWBitKYWaiAqY/46ZGSuiqqa2dom2nRS8OQGSgjgBkoI4AZKCOAGSgjgBkuLHMUwvZOybb/VU4/X6VfF47zZRu2SW2bB9RBIsBpOtdVnGw6Azofu9zoTuTMZzuy1nSd3m8Mpcf2VS4oMY/RARsRHXuN3pSd8Hc/Ndz5zDXp9TfbeRmdhdd6aXkelzFL1+Nx1qfY/3lZq+rT/LZ3LFmndeAQDvBcQJkBTECZAUxAmQFMQJkBTECZAUa6X88z/9g4w1tdmo3pQ3DTdiInBERGM2vjcj3YOnaUwrftXa37bv1xueR2aMgBtbUFX6u3Win1E70Wv2B9Ofx9gUvbF0DsLLOuzMRnqzGb3v9XU4K6uqytfRGyul7/V17LfGWppo62NU6eeg6SbF4+1Iv+vM4yHhzQmQFMQJkBTECZAUxAmQFMQJkBTECZAUm+D9x7//axmrBq3r0aicah4ZK6Wu9KXUZnyCm5LcjsuWSTc2/WHGelJ2NylPyo6I6Dodazt9zpGwgmrTF6eqfoDKGVF94tZYS8T87TfGFulEz5+xGWkxm+nnamZsuLGw/CIiukbfx1GUrZvaVDRVe6pSAD4YECdAUhAnQFIQJ0BSECdAUhAnQFKslTLXDkD0O92AqmnKlQVNY6opHnRlwcO9bt9vG1DJJlOmYVilU++VsIgiIuqRtlLqVseqkargMZUzzj4yVTpuurKarNC0ek030dcxm2qb4nSu7+N0Vr5Xl2bN+amOzef6fpjipBi1unnZdFy+WZOZq5DSdo+CNydAUhAnQFIQJ0BSECdAUhAnQFIQJ0BSrJVisuFRm2A3Lqe2ZyJNHhHRmP+Juxs9l+XGxFb3q+Lx3jStCjEHIyJi1On0+nimK2e6xVR/XleO3a61VbXdPchYH3pdZ2yi2bic6j+d69/54lT/nk+WpzJ2ZWLnp2X/bjE2c0gGbaftB23RNaZ0Zj7V322xXJTXLPSasdCEgzcnQFIQJ0BSECdAUhAnQFIQJ0BSbLb25sULGWtGJjvZlTN8+8mdXDMTayIiarO5fRQ6u9rW5exqNdJZVzXCISLiXGTpIiI++fhjGfvyyz+XsS8+/0rGFLdvdYbaTdF2/8Sd2Pk+MSMGJqbP0Ujc+wg/TuJ+U56KvlqVJ4BHRIToOxQRMVue6NjZuYwtlmcydnJWzjbPT/TzoTTh4M0JkBTECZAUxAmQFMQJkBTECZAUxAmQFGulVKE3iN/f6r4+D8NteU1t7BezCbkyPX92ZvzAVowLGMz5GtNSf1iZkRFv9Ob2+Qu9IfpTsVn6xxdX+rNmehP1ttP3cWNsltWmvJn+5m25eCAi4tXaxDY6tjJWymEqRmiYzfJzY4lMl2YzuilIaM09bqfldU2r+wRVR4y25s0JkBTECZAUxAmQFMQJkBTECZAUxAmQFJvfnZ3oeQybje5Vs1uLKcmmP8/O2BvOShnCVD8MoirFTGNwk5zd6IfVSltLb57r6p6XX/+2ePzpTFdFdOZ+3Ia2gt5U2nb6o/htXtT6fDedtsZ60QsoImJ8uZSxxdNybH6u7ZLx3EwVnxhLxEw47ybaZpGWibEKBzfqW8CbEyApiBMgKYgTICmIEyApiBMgKYgTICl+HMNMp8PvOl11sBFWykFYGxHa9ogIO4jaOCky6P6RTFFKHHrT9t9WzujYg3Bu7syk7zBp+e+MXfVtp6/jxbT8KGzPtKXQXermWYsnFzI2O1vK2GRRbpLlGso5b6wyTcjcOZvm3WO1qTypzfnkmndeAQDvBcQJkBTECZAUxAmQFMQJkBTECZAUa6W0nZ6EXJsZGiGyxsNB+x4HZ5c4L8WFrM1SpjfezIOJvTaNxn53MLZIX65meWnmufStvvd/1D9ZrM0k6vF12fpYXutGYyfnunJmPDuiqiMiKmE5VM72aPVj3Ix0rKrNM2yfxzK1sXRqqlIAPhwQJ0BSECdAUhAnQFIQJ0BSECdAUvwAh0aXaDQ6Gx6TWTnt3ff6fMZlsTFnz1SqGsScrzfp8HvXGMyMWV+7Jllt2YL5/USvmZ/paqHuXFeKLJ/oJlkXF+XYYqHPNzJNsA6dfrQGU6FRtWUvqJ3oZlwTMbvk+2KdafDVGrtH2TPWmjkC3pwASUGcAElBnABJQZwASUGcAEmx2VqVdY2IOD3Xbe4n07Lm+53OQO7dpngTE8Or/ydYPuw2IbuM28hkGV2RwMxkGheiT9Ppqc6SXp7qDednp+UePBERMzPJeTwpZyeb1ozCMBlq1y/KFis05fvvevDUYk1ERO02zNseQuacpi+Rwo0NkZ/zzisA4L2AOAGSgjgBkoI4AZKCOAGSgjgBkmKtlOWFTsubPcOx35VHApih0eEy74MZZ1AZW6QWtkhj7BKXQm9bnXqfWCtFx07E5vFTMwpjPjFTozu9Ybs2m7kHYStsjd0wOEvB3EdrSYmRBm48wrEbzp1Ft9/rnlD7vhyrKmP30EMI4MMBcQIkBXECJAVxAiQFcQIkBXECJKUanIcBAH8yeHMCJAVxAiQFcQIkBXECJAVxAiQFcQIk5b8AfOkgiIdsEW0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAARZklEQVR4nO2dyY4sZxGFI6eq6urq2237AhLzJEBixYoX4D15D96DYYNkAcb4+rZ7rsrKkcVl+Z+D3AsIWd+3rNCfndPplOL8EVGt6xoAkI/6/30CAFAGcQIkBXECJAVxAiQFcQIkpXXB3//hjzKVuy6TXNfUVfH3w34v11xfv5Gx3aaTsa7R/19UrK10hrotn/p/jqdjTaOP2ZmD6og+3jLP+kTMut1mK2ONvI8mm2/ulV3nVoll66KP5xyHutInWdfu22SOKS68bvQLUpmbdXXYFoN8OQGSgjgBkoI4AZKCOAGSgjgBkoI4AZJirZSLrdHurGOVSENvjN2wa3Vs2ywy1tbGwqjK65TV8+F4MhS1sWCMyxJNpaO1OBdnD6yzvh/jNOqYsT7WpXyObePulY4tqz7HylgOIawPfbSIdTV2SeXeU3tQfUxx3e5+vAa+nABJQZwASUGcAElBnABJQZwASUGcAEmxVkobuvIkKl0ZUYvEd1cNcs2m1sfbmnS+sm0iImqRDm/M/6TKVD8siz7H1dgDi7FSlK2wuqqURRsLs6lYOU5HvU5YMK4iaNPp16cx1SC7na6OCWF9qEqQ/0ZlTRjNaqwgXTqj36vVPLOI8v3gywmQFMQJkBTECZAUxAmQFMQJkBSbrW1MtnZd9QbrRmwQb8328GY9y1htsqSmhVCoJG/jMn9mw/NiNpUPOhTLpG9zt9kUf6/MJup50s9lGHRG3CSiYxLX9vyiM7zuftxcHWSsc1le8Yq4wQSuv1Bj+gS5e7zM+h4vYqP9YgsLvn5PJb6cAElBnABJQZwASUGcAElBnABJQZwASfEb30UPnoiIZdWp5lps2q4XneZvjDVTm03ItekfozYbr27ju9mwPc/6/OdR349zL0OyZ05lLIC205vRJ+PpPDy/yNixL1tZbsTAxlgRl3vT58jcK1lcYN4B22+pNv2bTH8hN/JCvXPzagojsFIAvjkgToCkIE6ApCBOgKQgToCkIE6ApPiqFGMdrJOOhUgpr2asQj3v9HmYdH5r0uGrOI9KjB6I8FZKbaylMUzvnqP2Uk7CwujPukqnac1jM/djMBbGo6g+eff+vVzjpor/+le/kLH9XvcQ2nbimLO2iBYTc1UpjbFZXBWMGq/hqlIWMwk+4tvFX/lyAiQFcQIkBXECJAVxAiQFcQIkBXECJMVaKUuvqxiWs7FSxI7+2aS8540+lXWnqzCqjbFFRHVMZcY7aEMkYh5N8y8zbbppTSMpMS7g/Vf3cs2TabrVu05jrgpDVE18/q8v5Jrn50cZe/funYz97Effl7Gf/OB7xd8/ur6Sa1wjusZUirRuUrmxWVTMvFa2YVjEz4u/8uUESAriBEgK4gRICuIESAriBEgK4gRIirVSTs/PMjaYrlWjqLQ4mxkZGzMPZTZVDK4yQhWz1J2bQq3z4cOsU+/GZXHzwWMUs02qRt+r2Ty2uydtf93e3cvY/V3ZFplG09TMWGOff66tlL/86c8y9u2Prou///LnP5VrfvSD78rY5da8c/o1iMZMxFaT27etqXIxVSnf+/Vvxd8BgJQgToCkIE6ApCBOgKQgToCk2Gzt3/7+mYydB93jpj+dir9vTC+g27s7GXMZ2a7VscNFuS/RxmTw3BiERfSOiYhYTWw2xzyJvj7Pg85eP590BvXhRT+X49ls9N5eln/fXMg1i+kj9dHNjYwdtuZencsOwT8++6dc8/SoN+DvNrpoojPZ1cX0z9qIdRe78pTyiIjZTAH/ze/Kv/PlBEgK4gRICuIESAriBEgK4gRICuIESIq1Uv78109lbBX9eT4Ey7HFTCf+9N2XMtYaK0WltSMi9jthpZg1bvRD3ei0fN3qzfmrG5Egpm8vxn7pB72JundjtE2Tm21XtgEqswG8CdPbadHWwTCUrbaIiLc35Y3vn9zoHkKrea9eXnQhwP2XOvZkij6aTflZt9aacd2pyvDlBEgK4gRICuIESAriBEgK4gRICuIESIq1Ur540Onk7UbvwK+F9WHa80Sz6mBlevdsQq97OZYth8p09WlbYw80+jyqWsfmVf8PrERr/6rW9sAwaLvEVT80xkqpq3KqvzPW0tVeV6zs2r0+j0XHqqVcDfL0dC/XuGqQN2/K1TYREZNxA+9PurrnuRfjRox95CamK/hyAiQFcQIkBXECJAVxAiQFcQIkBXECJMVaKYOxAFZjb1yIpkrbra7c2HQ6Vhm7JEx1jKow2V7o9PpsLJ3BTCdujAXTNKbxk3BMzqOZUG1sltY0z+qMFaRGE+zNVPHDTj+zt2YS9Xc+LleeRERM5/LU7ttbPd7h4e4rGTuZZ9Yaa6kxFSbnU9nKmkU1VoR7SzV8OQGSgjgBkoI4AZKCOAGSgjgBkoI4AZJirZStmJ8REbEzVSmzmDNxMjv961ofzzVwCtM86+bqTfH3648+NsfTKfTTWc/PmCbTwMnNURETvVeTlp9MVY1b5xp8tduyZbLb6ueyMxPCd8aKaI0z1or36uZK2y/9s27U9fykK6vOyseKiOlUtnQiIo6q+Zdo/BURsVKVAvDNAXECJAVxAiQFcQIkBXECJAVxAiTFWikXe11Z4EbBT2M5je7mRfS9qfgQczwiIvaX2u5ZqvLlnc664uPiQv+tw0Hfj8FYKeui7Y1azESprkxjqlE3+FpMk6m20tbBRvgblxv9ilxutD2w2+p1i2lCNohZL4N5ZtOkr+vBWCmjsVIWUxU0vJSPOfX6udSdru6Ra772CgD4n4A4AZKCOAGSgjgBkoI4AZJis7W7i4OMmT3Usb8sZzXbxv05vVHabW7fiOnVERGHq/L57/d6HEBnspON2eitxipEhJz0HRGxqIyh2BAfEVGbadO12RQ/j3qi9DqVYxuT4TW3I2pzzW5khNof3rT6uWx2ZrxDq7PvD/fvZezlqDOvnXgda9MpqHLFGwK+nABJQZwASUGcAElBnABJQZwASUGcAEmxVsr+UO7BExGxNRt5t135sG5q9IWxbbYmVV6bFHurzsN4ALW7I7YNjE6Vr6vpLySKAebR2C+jtkuqRZ+HS+ePQ7k/0jjo/jynRZ9Hbza+Xxr7ayt6FpnagdibgoQf/vgnMrbb6+KCOzPi4WNR5GB7ErkLEPDlBEgK4gRICuIESAriBEgK4gRICuIESIq1UppGt5dXYwQiIiaRznd9h2rTvr9u9TpXraC8j9lM5Z6N3bCaCo11NSMS3PiEuVyhUZnKDbUmIqIyfZpWN5lb2ET9K/vsdG4cg7n/07l8jqvxsVysNtVC33r7VsZ++uMfylgl+j598eWtXPP0pC0pBV9OgKQgToCkIE6ApCBOgKQgToCkIE6ApFgr5fmoG0K5icd1XU6VPz49yTWPpm3+4Y2ujnlzcyNjTVOugqlMkYirLnGxedVTr+dFx0KMT6hWY5dM2hLpam0rXOz0M2vFGIoL85ydpeOslMXYcKOwgiZzzcNZT0xfzDm25l6Fml4dEa2w7y5ctc1GxxR8OQGSgjgBkoI4AZKCOAGSgjgBkoI4AZJirRTXz2o0TaamsWw5NKa6xKWh3ayR/qR3+3cbYQOYf0lujsdqYmGaeFXGgunEROnONCHrdmaei3to5j6qISVtpyuTFjPDZjXVPa6iqRcTrI/G1hsGbaWce73udNTvjimgiu22fE+2uwu9BisF4JsD4gRICuIESAriBEgK4gRIis3WTqbnzDzojNuVmCh9Yzap73Y6Kxgm87eYnjnjUJ5OvJjs6RJmorTJyLpJzm5Ewiha+0+9Pt5oUrKdGEER4ScvN2oTuDl3NxZiGvVm/3Ovp0b353J2Vf0eEfFiNqk/3N/L2MlkgE8myzuLZ3Z9fS3XHK70yAgFX06ApCBOgKQgToCkIE6ApCBOgKQgToCkWCvl+hPdrt7ZGyotP5qN49OzTr1XxgKw9oCwHDZbM2Hb9NlxE6qH/ihjj48PMta/lDdfD2dtNyxmorTr7XQ46AnhV5flKc+V2Svveve4zegu9nIs2yJ393rStLNSZjMy4nzU9/jlxRxTWCm3X+lzdJafgi8nQFIQJ0BSECdAUhAnQFIQJ0BSECdAUqyV4qRbmf4xk0gbj4NOay/GZqmMbeOsFLXOVW7sL3UfGNcDydlEs6nsOIkKjbvb93LNIKptIrR9FKGrhSIi3n7ySfl4YopzRMTZ2D39SVd1PD7qsRzKSjmba56MXTKYCpizOccwE72VPfNkxo0420nBlxMgKYgTICmIEyApiBMgKYgTICmIEyAp1kp5fLiVsTdXetq0qiIZJ1150h91VcdoqhimUcdUyrsRowci/CTny4O2WQ6HclVHRETX6Nu86coVMp34PcJXxxiHy9o9t1+Vn7WyxSIiejvqwD1PU50kJlg728Y1DHNWymoa2K2i8uTDMcvXPQz6/V5MdY+CLydAUhAnQFIQJ0BSECdAUhAnQFIQJ0BSrJUyHs0uezGROSKirstTmV3lybLoWH/SzZaenx9lTE1y3hqbwlV8nAdtHYwm1e8qRUYxU6Rt9aNZFh2bjF11fNG204N4NpN7ZjISsZrZMaOxHFSzrrHX574an2I2VtDsZr0Y+26RNot+zsa9k/DlBEgK4gRICuIESAriBEgK4gRIis3WDmaa8OnZTUkuZ2tXk82qTe6vbfT/kFZNZI6IdSln1ZbV9Csy5zFNOnZ/r++V6x/Tiyyvy5Kupm+SG5NhY+KYi9lkfzIZVHVdH05Dn0clQk1VfqciIobZFUaYbK15nuuq37lKvN+1eb9N8lrClxMgKYgTICmIEyApiBMgKYgTICmIEyAp1kq5NyMB+t1OxhphfbgUuotNsxl1YDfTlzc2z+Z402Q2PAtr5kPQjIUwjX1qda/0X7Ln4Tacu3NUIw1cIYAbg+Dw74E4R3NdznZShQUfDunuso6ta9kyqULbPWaqhV7z9ZcAwP8CxAmQFMQJkBTECZAUxAmQFMQJkBRrpfzrs0/1QtPjphKVItYCMOl1h+vNUolgXes0uU95G7vEVdyYg1azTr8rFlNdMotxBhER/cn14Sk/G1P0E6uZ/ryY3j2T6d2zimtztoet4LHvlTl/s05dWy0slgh9XQ6+nABJQZwASUGcAElBnABJQZwASUGcAEnxk63ff/6qg6reSE2jbYOqNZUboqHSfztmIyZKq6qZD3/LtdR3qXJTxWAsB7XKWUTOmnH3atOZx72U1w2mUZcbq+BGHdh7pRqNWWvDNeoylUTG3nCjQ1ZhpYxmUrazuBR8OQGSgjgBkoI4AZKCOAGSgjgBkoI4AZJirZTazBRZTDq8EqHFzZ8wzZFcqci66NgiKj4mdzzb8+kVAy8iojF/rxVWUOXKY9x5uOdS68etLIzxrO0SNwOmfqXtNAvLZDUVJK7iww2UXoz1sZjGYItoEOdmC01YKQDfHBAnQFIQJ0BSECdAUhAnQFIQJ0BSrJXi0teuQkPlr+0aOxzkdWPWld1TGfvFzTWxp29OcTYp+1Wk2N3fakywFpU4H2JmBLuo7HCVJ2EqRUwtiK0wkVUp1i4xVS5uBo+b9WKa0U1ifsy5P77qPBR8OQGSgjgBkoI4AZKCOAGSgjgBkmKztWEzl3YOQvFn19/GjixwMZs0LgdVRjDCj4yYzYgB1yNmtmMoREZZrohozXNpu07Htnoauco2uyyjzbqaDLufJ12OumemRklERMxusrVZ53oIjWrDv7kf7nkq+HICJAVxAiQFcQIkBXECJAVxAiQFcQIkxfcQMpuonb1Rq1S/8T1cqtmmoW1bH5HaNpuoV9tXRscmk7J3G6znqZzOdxaGs526jbZLNrO+WU1TtmDchGo3BsHymgfqRji4vknmL7lrG3o9hkJP0n7dKA8FX06ApCBOgKQgToCkIE6ApCBOgKQgToCkVC4NDQD/P/hyAiQFcQIkBXECJAVxAiQFcQIkBXECJOXfiII0D1GUD2sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show(X_corr[0])\n",
    "show(X_corr[1])\n",
    "show(X_corr[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send these examples to the predictor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c273cacad533435c87eeb07346578750",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=50.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "for i in tqdm(range(0, 5000, 100)):\n",
    "    X = X_corr[i : i + 100]\n",
    "    predict(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now when we check the message dump we should find a new drift response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Drift True\n"
     ]
    }
   ],
   "source": [
    "res = !kubectl logs -n cifar10 $(kubectl get pod -n cifar10 -l app=hello-display -o jsonpath='{.items[0].metadata.name}')\n",
    "data = []\n",
    "for i in range(0, len(res)):\n",
    "    if res[i] == \"Data,\":\n",
    "        data.append(res[i + 1])\n",
    "j = json.loads(json.loads(data[-1]))\n",
    "print(\"Drift\", j[\"data\"][\"is_drift\"] == 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tear Down"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace \"cifar10\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete ns cifar10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
